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Abstract 

To provide generalized solutions if a given problem admits no actual solution is an 
important task in mathematics and the natural sciences. It has a rich history dating 
back to the early 19th century when Carl Friedrich Gauss developed the method of 
least squares of a system of linear equations — its solutions can be viewed as fixed 
points of averaged projections onto hyperplanes. A powerful generalization of this 
problem is to find fixed points of averaged resolvents (i.e., firmly nonexpansive map- 
pings). 

This paper concerns the relationship between the set of fixed points of averaged 
resolvents and certain fixed point sets of compositions of resolvents. It partially ex- 
tends recent work for two mappings on a question of C. Byrne. The analysis suggests 
a reformulation in a product space. 

Furthermore, two new algorithms are presented. A complete convergence proof 
that is based on averaged mappings is provided for the first algorithm. The second 
algorithm, which currently has no convergence proof, iterates a mapping that is not 
even nonexpansive. Numerical experiments indicate the potential of these algorithms 
when compared to iterating the average of the resolvents. 
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1 Introduction 

Throughout this paper. 




and induced norm || • || . We impose that X 7^ {0}. To motivate the results of this paper, let 
us assume that Ci, . . .,Cm are finitely many nonempty closed convex subsets of X, with 
projections (nearest point mappings) Pi, ... , Pm- Many problems in mathematics and the 
physical sciences can be recast as the convex feasibility problem of finding a point in the 
intersection Ci Ci ■ ■ ■ Ci Cm- How^ever, in applications it may well be that this intersection 
is empty. In this case, a powerful and very useful generalization of the intersection is the 
set of fixed points of the operator 

(2) X^X:x^ ^^"+---+^-\ 

m 



(See, e.g., uM ^or applications.) Indeed, these fixed points are precisely the minimizers of 



the convex function 

m 

(3) X^IR: x^Y^\\x-Pixf 

i=l 

and — when each C, is a suitably described hyperplane — there is a well known connection 
to the set of least squares solutions in the sense of linear algebra (see Appendix A). 

A problem open for a long time is to find precise relationships between the fixed points 
of the operator defined in ^ and the fixed points of the composition Pm o ■ ■ ■ o P2 o Pi 
when the intersection Ci n • • • n is empty. (It is well known that both fixed points sets 
coincide with Ci n • • • n provided this intersection is nonempty.) This problem was re- 
cently explicitly stated and nicely discussed in [TTl Chapter 50] and |[T2l Open Question 2 
on page 101 in Subsection 8.3.2]. For other related wor see im, m, IIT3I , IIT6I , and the 



^In passing, we mention that when Ci, C2, C3 are line segments forming a triangle in the Euclidean plane, 
then the minimizer of (|3j is known as the symmedian point (also known as the Grebe-Lemoine point) of 
the given triangle; see l20l Theorem 349 on page 216]. 
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references therein. When m = 2, the recent work pO|| contains some precise relationships. 
For instance, the results in |j30l Section 3] show that 



(4) Fix(P2 o Pi) ^ Fix (iPi + iPz) : x ^ + iPix 
is a well defined bijection. 

Our goal in this paper is two-fold. First, we wish to find a suitable extension to describe these 
fixed point sets when m > 3. Second, we build on these insights to obtain algorithms for finding 
these fixed points. 

The results provided are somewhat surprising. While we completely generalize some 
of the two-set work from 113011 , the generalized intersection is not formulated as the fixed 
point set of a simple composition, but rather as the fixed point set of a more compli- 
cated operator described in a product space. Nonetheless, the geometric insight obtained 
will turn out to be quite useful in the design of new algorithms that show better conver- 
gence properties when compared to straight iteration of the averaged projection operator. 
Furthermore, the results actually hold for very general firmly nonexpansive operators — 
equivalently, resolvents of maximally monotone operators — although the optimization- 
based interpretation as a set of minimizers analogous to (H) is then unavailable. 

The paper is organized as follows. In the remainder of this introductory section, we 
describe some central notions fundamental to our analysis. The main result of Section |2] 
is Theorem |ZT] where we provide a precise correspondence between the fixed point set of 
an averged resolvent }a and a certain set S in a product space. In Section |3l it is shown 
that S is in fact the fixed point set of an averaged mapping (see Corollary l3.8|) . This insight 
is brought to good use in Section HI, where we design a new algorithm for finding a point 
in S (and hence in Fix and where we provide a rigorous convergence proof. Akin to 
the Gauss-Seidel variant of the Jacobi iteration in numerical linear algebra, we propose 
another new algorithm. Numerical experiments illustrate that this heuristic algorithm 
performs very well; however, it still lacks a rigorous proof of convergence. An appendix 
concludes the paper. The first part of the appendix connects fixed points of averages of 
projections onto hyperplanes to classical least squares solutions, while the second part 
contains some more technical observations regarding the heuristic method. The notation 
we utilize is standard and as in ||3]], (U, IIZSll , ||26| , ||28| , l|29ll , or ||3T1 to which we also refer 
for background. 

Recall that a mapping 

(5) T:X^X 

is firmly nonexpansive (see ||32| for the first systematic study) if 

(6) (VxGX)(VyGX) ||Tx- Tyf + ||(Id-r)x- (Id -T)yf < ||x-y||2. 
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where Id: X — > X: x i— ^ x denotes the identity operator. The prime example of firmly 
nonexpansive mappings are projection operators (also known as nearest point map- 
pings) with respect to nonemtpy closed convex subsets of X. It is clear that if T is firmly 
nonexpansive, then it is nonexpansive, i.e., Lipschitz continuous with constant 1, 

(7) (VxGX)(VyGX) \\Tx - Ty\\ < \\x - y\\; 

the converse, however, is false (consider — Id). The set of fixed points of T is 

(8) ¥ixT = {x eX\x = Tx}. 

The following characterization of firm nonexpansiveness is well known and will be 
used repeatedly. 

Fact 1.1 (See, e.g., ||3HT8HT9l.) Let T: X ^ X. Then the following are equivalent: 

(i) T is firmly nonexpansive. 

(ii) Id —T is firmly nonexpansive. 

(iii) 2T — Id is nonexpansive. 

(iv) (Vx G X)(Vy G X) ||rx-ry||2 < (x-y,Tx-Ty). 

(v) (Vx G X)(Vy G X) < (Tx - Ty, (Id -T)x - (Id -T)y). 

Firmly nonexpansive mappings are also intimately tied with maximally monotone op- 
erators. Recall that a set-valued operator A: X ^ X (i.e., (Vx G X) Ax C X) with graph 
gr A is monotone if 

(9) (V(x, u) G gr A) (V(y, z;) G gr A) {x-y,u-v)> 0, 

and that A is maximally monotone if it is monotone and every proper extension of A fails 
to be monotone. We write dom A = {x G X | Ax 7^ 0} and ran A = A(X) = IJ^^x^^ 
for the domain and range of A, respectively. The inverse of A is defined via gr A~^ = 
{(m, x) G X X X I u G Ax}. Monotone operators are ubiquitous in modern analysis and 
optimization; see, e.g., the books 0, 0, 0, 113, lH, BH, ill, BH, il, and iS. 



Two key examples of maximally monotone operators are continuous linear monotone 
operators and subdifferential operators (in the sense of convex analysis) of functions that 
are convex, lower semicontinuous, and proper. 

Now let A : X =^ X be maximally monotone and denote the associated resolvent by 
(10) /^=(Id+A)-i. 
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In ||23|, Minty made the seminal observation that }a is in fact a firmly nonexpansive op- 
erator from X to X and that, conversely, every firmly nonexpansive operator arises this 
way: 

Fact 1.2 (Minty) (See, e.g., 1231 or |[T7||.) Let T: X — ^ X be firmly nonexpansive, and let 
A: X ^ Xbe maximally monotone. Then the following hold. 

(i) B = — Id zs maximally monotone (and Jg = T). 

(ii) J A is firmly nonexpansive (and A = — Id). 

One of the motivations to study the correspondence between firmly nonexpansive 
mappings and maximally monotone operators is the very useful correspondence 

(11) A-\Q)=Vix]A, 
where A: X ^ X is maximally monotone. 



From now on we assume that 



(12) Ai,.. 


. , Am are maximally monotone operators on X, where m G {2, 3, . . .}, 


that 


(13) Ai, 


. . . , A„, belong to ]0, 1 [ such that ^ A; = 1, where I = {1,2, ... ,m}, 




iel 


and we set 


(14) 


A=(^XjA)j -Id. 







Then the definition of the resolvent yields 

(15) jA = E^rJA,-' 

iel 



thus, since it is easy to see that Ja is firmly nonexpansive, it follows from Fact 11.21 that 
A is maximally monotone. We refer to the operator A as the resolvent average of the 
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maximally monotone operators Ai, . . . , Am and we note that }a is the weighted average 
of the resolvents }a- The operator is the announced generalization of the averaged 
projection operator considered in (|2]), and Fix is the generalization of the minimizers of 
the function in ©. 

This introductory section is now complete. In the next section, we shall derive an alter- 
native description of Fix 



2 The Fixed Point Set Viewed in a Product Space 

It will be quite convenient to define numbers complementary to the convex coefficients 
fixed in (|T3l) ; thus, we let 



(16) j-ii = 1 — A,, for every i G I. 



Several of the results will be formulated in the Hilbert product space 



(17) X = X'", with iimer product (x,y) = ^ {xi,yi) , 

iel 



where x = and y = {yi)iei are generic vectors in X. The set S, defined by 




turns out to be fundamental in describing Fix 

Theorem 2.1 (correspondence between S and Fix /^) The operator 

(19) L: S ^ Fix/^: x = (x,),^: ^ J^^iXi 

iel 

is well defined, bijective, and Lipschitz continuous with constant 1. Furthermore, the inverse 
operator ofL satisfies 

(20) L-i: Fix/^^S:x^ (Ja.x)^^, 
and is Lipschitz continuous with constant ^/m. 



Proof. We proceed along several steps. 

Claim 1 : (Vx G S) Lx G Fix and x = (/^.Lx) consequently, L is well defined. 



Let X = {xi)i^i G S and set x = E/gj A/x, = Lx. Using the definition of the resolvent, 
we have, for every i G I, 

(21a) ^ — ^Xy G (ld+//j~^A,)x, ^ AyXy G f/fXf + A^X/ 

(21b) <^ ^ AjXj G (1 - A,)x, + A,x, 

iei-^{i} 

(21c) <^ X = ^ AyXy G ( Id + A,) X; 

(21d) ^ X; = /^,x = JaiLx. 

Hence x = (j/^Xx) as claimed. Moreover, (Vz G 7) A/X; = XiJ^.x, which, after sum- 
ming over i G I and recalling ((T5|) , yields x = E/ejA/x, = YLiei^iJAiX = }a^- Thus 
Lx = X G Fix and Claim 1 is verified. 

Claim 2: (Vx G Fix/^) (/a,-^);^; ^ S. 
Assume that x G Fix }a and set (Vz G I) yi = Jai^- Then, using (|T5|) , we see that 



(22) Y^\^y- = Y^\J^^x = fAX = X. 

iel iel 

Furthermore, for every i G I, and using ^2) in the derivation of (|23c|) 

(23a) = /^^x <^ x G + A^y, x - A,y, G f^^y, + A,y, 

(23b) ^ (x - A;y;) G ( Id +^i'^Ai^ yi 

(23c) ^^-1 E A^y^ G (id+^-iA^jy^- 

yeixlO 



(23d) ^ y,- = / - 



Thus, {yi)iei £ S and Claim 2 is verified. 



Having verified the two claims above, we now turn to proving the statements an- 
nounced. 

First, let x G Fix J a- By Claim 2, (/A,x),ej G S. Hence L{fAiX)i^j = Eiei^^iJAi^ = Ja^ = 
X by (|T5l) . Thus, L is surjective. 



Second, assume that x = (x;)/^/ and y = {yi)iei belong to S and that Lx = Ly. Then, 
using Claim 1, we see that x = (/^.Lx);^; = (/^.Ly);^/ = y and thus L is injective. 
Altogether, this shows that L is bijective and we also obtain the formula for L~^. 



Third, again let x = (x,);^/ and y = {yi)iei be in S. Using the convexity of 
obtain 



-, we 



(24a) 
(24b) 



I Lx — Ly I 



Y^Xj{xi-yi) <Y^:ii\\xi-yi 

iel iel 



<J^\\xi-yi\ 

iel 



|x-y| 



Thus, L is Lipschitz continuous with constant 1. 

Finally, let x and y be in Fix Since Ja^ is (firmly) nonexpansive for all i G I, we 
estimate 



(25a) 
(25b) 



iL-^x- L-^ 



l.,l|2 



iel 



< 



m\\x — y\ 



iel 



Therefore, L ^ is Lipschitz continuous with constant \/rn. 
Remark 2.2 Some comments regarding Theorem 12. II are in order. 



(i) Because of the simplicity of the bijection L provided in Theorem 12.11 the task of 
finding Fix is essentially the same as finding S. 

(ii) Note that when each A, is a normal cone operator Nc,, then the resolvents and 
^ir^A- sin^plify to the projections Pq, for every z G 7. 

(iii) When m = 2, the set S turns into 

(26) S = {(xi,X2) G X I xi = /;^-i^jX2 and X2 = /a-Mj^i}' 

and Theorem 12. 1 1 coincides with pOl Theorem 3.6]. Note that (xi, X2) G S if and only 
if X2 G Fix (/a-ia2^a ^Aj) ^^'^ ■^'^ ~ Mi'*^2/ which makes the connection between 
the fixed point set of the composition of the two resolvents and S. It appears that 
this is a particularity of the case m = 2; it seems that there is no simple connection 
between fixed points of L, 1 . [„ 1 . • • • [,,1 . and Fix J a when m > 3. 
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3 Fixed Points of a Composition 



From now on, we let 




and 



(28) J: X ^ X: X = (xO.el ^ (V^^O^e/ 



It is immediate from the definition of the set S (see (|T8)) ) that 

(29) S = Fix (J o R). 

We are thus ultimately interested in developing algorithms for finding a fixed point of 
J o R. We start by collecting relevant information about the operator R. 

Proposition 3.1 The adjoint of R zs given by 

(30) R*:X^X:x=(xO,el^ f E -^i) 

^ /€7\{/} n / iel 

and the set of fixed -points o/R zs the "diagonal" in X, i.e., 

(31) FixR = {(x),ej G X I X G X}. 

Proof. Denote the operator defined in (|30l) by L, and take x = (X;)/^/ and y = {yi)iei in X. 
Then 

(32a) (x,Ly) = E(x.(Ly),>=E E ^(^"^7) 

iel ieljel^{i} n 

(32b) = E ^.(^-yj) 

(32c) =E E ^(^^'y/> = E((R'^)7'yi> 

jeliel\{j}rj jel 



(32d) =(Rx,y), 
which shows that R* = L as claimed. 
Next, let X G X and denote the right side of (|3T]) by A. Since 

(33) (VzGl) ?^-% = l, 

it is clear that 

(34) A C Fix R. 

Now let X = (X;),g7 G FixR and set x = E^e/ A/x,. Then x = Rx, i.e., for every i G I, we 
have 

(35a) Xj = {x)i = (Rx); <^ Xi = ^ — ^xy <^ }iiXi = ^ AyXy 

(35b) (1 — A,)X; = ^ AyXy X, = X^AyXy 

/6J\{/} ;ei 

(35c) ^ Xi = X 

by (|T6)) . Hence x = {x)i^i G A and thus 

(36) FixR C A. 

Combining (|34|) and (|36l) , we obtain (|3T1) . ■ 



Remark 3.2 If m = 2, then R* = R. However, when m > 3, one has the equivalence 

R* = R ^ (A0.67 = (i).67- 

The following observation will be useful when discussing nonexpansiveness of R. 
Lemma 3.3 We have 1 < mY^?^; furthermore, equality holds if and only if (Vz G I) A, = ^. 

Proo/. Indeed, 

(37) 1 = E^r 1 < (E^?)'''(El')''' ^ 1 = 1' < iYl^y 

iel iel /6-f /6-f 

and the result follows from the Cauchy-Schwarz inequality and its characterization of 
equality. ■ 

The next result is surprising as it shows that the actual values of the convex parameters 



A; matter when m > 3. 
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Proposition 3.4 (nonexpansiveness of R) The following hold. 



(i) Ifm = 2, then R : (xi, X2) ^ {x2, Xi); thus, R is an isometry and nonexpansive. 
(11) Ifm > 3, then: R is nonexpansive if and only if i^i G /) A, = in which case ||R|| = 1. 



Proof. (1) When m = 2, we have Ai = arid A2 = thus, the definition of R (see 
(|27l) ) yields the announced formula and It Is clear that then R Is an Isometry and hence 
nonexpansive. 



(ii)j Suppose that m > 3. Assume first that (Vz G 7) A; = ^; hence, = 1 — ^ 



(m — l)/m. Then 
(38) 



1. 



Now let X = {xi)i(^i G X. Using the definition of R (see (|27|l), the convexity of || • |p In 



(|39b)l . and dSS]) In (l39e)) . we obtain 



(39a) 
(39b) 
(39c) 

(39d) 
(39e) 
(39f) 



iRxl 



A; 



= EII(RxMI =E E i-, 

iel iel jel-^{i} 

<E E ^W-if 

{ai)eixl I i^i] 

= L^,\\-jf E ^ 

jel iel^j} 
= Ell-;f 

Il-,ll2 



Since R Is linear. It follows that R Is nonexpansive; furthermore, since FlxR 7^ {0} by 



(|3T|) , we then have ||R|| = 1. 



To prove the remaining Implication, we demonstrate the contraposltlve and thus as- 
sume that 



(40) 
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Take w G X such that ||u|| =1 and set (Vz G I) Xi = }iiU and x = (x/)i6i. We compute 



(41a) 




l|xf = 


Y^W^if = ]2\\nM\^ = 

iel iel 


iel 


(41b) 






E(i - = 

iel 




(41c) 






m-2 + J]a2. 

iel 




Using dSO]), the fact that | 


u\\ = 1, we obtain 






(42a) 


||R*xf = 


iel 


jel-^{i} 


2 

= EA? 

iel 


E V/" 

/€7x{f} 


(42b) 




iel 


{m — 1)m||' = 


iel 


Altogether, 












(43a) 


||R* 




X ^ = 2-m+ ((m-1) 


^-i)EA? 

iel 


(43b) 






= (m -2 




EA?)- 

/6l 



Now m > 3 implies that w — 2 > 0; furthermore, by (|40|) and Lemma I3.3i — 1 + 
mJ^i^iXf > 0. Therefore, 



(44) 



|R*x|| > ||x||. 



This implies ||R*|| > 1 and hence ||R|| > 1 by ||2Tl Theorem 3.9-2]. Since R is linear, it 
cannot be nonexpansive. ■ 

For algorithmic purposes, nonexpansiveness is a desirable property but it does not 
guarantee the convergence of the iterates to a fixed point (consider, e.g., — Id). The very 
useful notion of an averaged mapping, which is intermediate between nonexpansiveness 
and firm nonexpansiveness, was introduced by Baillon, Bruck, and Reich in (ll. 

Definition 3.5 (averaged mapping) Let T : X ^ X. Then T is averaged if there exist a non- 
expansive mapping N : X — > X and a G [0, 1 [ such that 

(45) T = {1- ix)ld+(xN; 

if we wish to emphasis the constant a, we say that T is a-averaged. 
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It is clear from the definition that every averaged mapping is nonexpansive; the con- 
verse, however, is false: indeed, — Id is nonexpansive, but not averaged. It follows from 
Fact II .11 that every firmly nonexpansive mapping is ^-averaged. 

The class of averaged mappings is closed under compositions; this is not true for firmly 
nonexpansive mappings: e.g., consider two projections onto two lines that meet at at a 
nl 4 angle. Let us record the following well known key properties. 

Fact 3.6 Let T, T\, and T2 he mappings from X to X, let dl\ and 0L2 he in [0, 1[, and let Xq G X. 
Then the following hold. 

(i) T is firmly nonexpansive if and only ifT is ^-averaged. 

(ii) IfTi is xi-averaged and T2 is oc2-averaged, then Ti o T2 is x-averaged, where 

0, ifci\ = (X2 = 0; 



(46) a 



otherwise 



1 + 1/ max{a.i, 0.2}' 
is the harmonic mean ofl and max{ft;i,ft;2}. 

(iii) IfTi and T2 are averaged, and Fix(Ti 0T2) ^ 0, then Fix(ri 0T2) = Fix(ri) n Fix (72). 

(iv) IfT is averaged and Fix T 7^ 0, then the sequence of iterates {V'^xo)neN converges weakl-jj^ 
to a point in Fix T; otherwise, \\T"xo\\ — > +00. 



Proof, (i) This is well known and immediate from Fact ll.li 



(ii) The fact that the composition of averaged mappings is again averaged is well 
known and implicit in the proof of (ll Corollary 2.4]. For the exact constants, see IITSl 
Lemma 2.2] or |j3i Proposition 4.32]. 



(iii) This follows from ^ Proposition 1.1, Proposition 2.1, and Lemma 2.1]. See also 
Theorem 3] for the case when Fix T 7^ 0. 



(iv) This follows from HI Corollary 1.3 and Corollary 1.4]. 



Theorem 3.7 (averagedness of R) The following hold. 

(i) Ifm = 2, then R zs not averaged. 

(ii) Ifm > 3 and (Vz G I) A; = ^, then R = (1 — a) Id +(xN, where oi = 2m-'i ^ ^'^ 
isometry; in particular, R is a-averaged. 



^When T is firmly nonexpansive, the weak convergence goes back at least to |[8l. 
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Proof. ^\ Assume that m = 2. By Proposition I3.4|ri)t R: (xi,X2) ^ {x2,xi). We argue 
by contradiction and thus assume that R is averaged, i.e., there exist a nonexpansive 
mapping N : X ^ X and a G [0, 1 [ such that R = (1 — a) Id +aN. Since R ^ Id, it is clear 
that a > 0. Thus, 

(47) N : X ^ X : (xi, X2) ^ ol~^ [x2 — Xi + OiXi, Xi — X2 + ^^^2) ■ 

Now take 1/ G X such that = 1 and set x = (xi,X2) = (0,ft;i/). Then 



(48) 

and Nx = [u, (a — \)u). Thus, 



loir + Wdcu 



|2 2 

= a 



(49) 



|Nx| 



|uf + ||(a = 1 + (1 = a;2 + 2(l - a) > 



Hence ||N|| > 1 and, since N is linear, N cannot be nonexpansive. This contradiction 



completes the proof of [(i) 



(ii)j Assume that m > 3 and that (Vz G 7) A, = ^. For future reference, we observe that 

1 



(50) 

We start by defining 
(51) 

Then it is easily verified that 
(52) 

and hence that 
(53) 

Now set 
(54) 



(V/gJ)(V;g7) ^ 



1 

m 



1-1 m-1 

m 



iel 



L* : X ^ X: X (^)j67 



L*LL*L = mL*L. 



m 



a 



2m -2 



and N = a-i(R- (1 -a)Id). 



Then a G ]0, 1[ and R = aN + (1 — a) Id; thus, it suffices to show that N is an isometry. 
Note that 



(55) 



cc-l 



m — 1 
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Take x = {xi)i^i G X. Using (|27|) , ((50|), and (|55l), we obtain for every z G 7, 



(56a) (Nx); = ( - (1 - + (Rx),) 

(56b) = a-'^({a-l)xi+ ^x^) 

/I 1 
(56c) = a.~^ I — ax, H + y x,- 

(56d) = a"^ ( - ax, + V ^ x. ) 

(56e) = — X,- H -Lx 

m — 1 

(56f) = -X, + — Lx; 

m 



hence, Nx = -x + ;^L*Lx. It follows that N = - Id +^L*L and thus N* = N. Using (|53|). 
we now obtain 

(57a) N*N = NN = ( - Id +|L*L) ( - Id +|L*L) 

(57b) = Id -|L*L - |L*L + ^ {VLVL) 

(57c) = Id -lrL+^(mL*L) 

(57d) = Id . 

Therefore, ||Nx|p = (Nx, Nx) = (x, N*Nx) = (x, x) = ||x|p and hence N is an isometry; 
in particular, N is nonexpansive and R is a; -averaged. ■ 

We are now in a position to describe the set S as the fixed point set of an averaged 
mapping. 

Corollary 3.8 Suppose that m > 3 and that (Vz G 7) A, = ^. Then J o R zs 3^-2 "^^^^^^'^^ ^'^^ 
Fix(J o R) = S. 

Proof. On the one hand, since J is clearly firmly nonexpansive, J is ^-averaged. On the 
other hand, by Theorem 13. 7[(ii)[ R is j^^^-averaged. Since < 5 < 2m -2 ' foUo'^s from 
Fact 13. ^(ii)1 that J o R is a-averaged, where 

2 2m 

(58) oi 



l + l/(m/(2m -2)) 3m -2' 
as claimed. To complete the proof, recall (|29|). 
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4 Two New Algorithms 



In Section|2l we saw that Fix = L(S) (see Theoreml2.1|), and in Section|3]we discovered 
that S = Fix (J o R) is the fixed point set of an averaged operator. This analysis leads to 
new algorithms for finding a point in Fix 

Theorem 4.1 Suppose that m > 3 and that (Vz G I) A; = ^. Let xq = (^o,j)f€-f ^ ^ '^^^ 
generate the sequence {xn)nef^ 

(59) (VnGN) x„+i = (JoR)x„. 

Then exactly one of the following holds. 

(i) Fix 7^ 0, (x„)„g]N converges weakly to a point x = {xi)i^i in S and (Ezei '^''^w^OweN 
converges weakly to Yliei ^i^i ^ ^^"^ Ja- 

(ii) Fix = and ||x„|| — > +oo. 

Proof. By Theorem |2.1l S 7^ if and only if Fix /^^ 7^ 0. Furthermore, Corollary 13. 81 shows 
that S = Fix(J o R), where J o R is averaged. The result thus follows from Fact I3.^(iv)[ 
Theorem 12.11 and the weak continuity of the operator L defined in (|T9|) . ■ 



Remark 4.2 The assumption that m > 3 in Theorem 14.11 is critical: indeed, suppose that 
m = 2. Then, by Proposition 13.^(1)1 R: (xi,X2) ^ (^2/^i)- Now assume further that 
Ai = A2 = 0. Then J = Id and hence J o R = R. Thus, if y and z are two distinct 
points in X and the sequence {xn)nen is generated by iterating J o R with a starting point 
xo = (i/,z), then 



(60) (Vn G N) x„ 



(i/,z), if n is even; 
(z,y), if n is odd. 



Consequently, {xn)nen is a bounded sequence that is not weakly convergent. On the 
other hand, keeping the assumption m = 2 but allowing again for general maximally 
monotone operators Ai and A2, and assuming that Fix 7^ 0, we observe that 

(61) JoRoJoR:X^X: (xi,X2) ^ (/a^ Mi/a-M/1' Wa^-^Ai^z) • 

Hence, by Il30l Theorem 5.3], the even iterates of J o R will converge weakly to point (xi, X2) 
with xi = /a-iai/a-ia2^i ^2 = Jx-^Aih^^A^^^- However, (xi,X2) ^ S in general. 
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Just as the Gauss-Seidel iteration can be viewed as a modification of the Jacobi iteration 
where new information is immediately utilized (see, e.g., 11271 Section 4.1]), we shall pro- 
pose a similar modification of the iteration of the operator J o R analyzed above. To this 
end, we introduce, for every G 7, the following operators from X to X: 





Xi, if i 7^ k; 


(62) (Vx = (xO,6i G X) (V/ G I) (Rkx), = < 


^Xj, if i = k. 


and 


(63) (Vx=(x,).6jeX)(V/G J) (Jfcx),= 


(xi, lii^k; 


It follows immediately from the definition of S (see ( 


HI)) that 


(64) S=nFix(JfcoRA 

kel 


)• 


This implies 




(65) S C Fix (Jm o R,„ o • • • o o Ri), 


and it motivates — ^but does not justify — to iterate the composition 


(66) T = J„, o R,„ o • • • o Ji 


o Ri 


in order to find points in S. 




Remark 4.3 In general, the composition T = J,^ o R,„ o • • • o o R^ is not nonexpansive: 
indeed, assume that (Vfc G J) A^t = so that J^t = Id. Then T = R^ o • • • o R^ and we 



show in Appendix B that this composition is not nonexpansive and neither is any Rj^. 

Remark 4.4 One may verify that o R^^ is Lipschitz continuous with constant i/m/ (m — 1) 
when (A,) = (^)jgj (see Appendix B). In turn, this implies that 

/ m \"'/2 

(67) T is Lipschitz continuous with constant 

Vm — 1/ 

Asm ^ +CX), the Lipschitz constant of T decreases to ^exp(l) 1.6487. 
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Remark 4.5 (numerical experiments) In our numerical experiments, we assumed that 
X = IR^*^, that m = 55, and that (A,),£j = (^)iej. We considered m hyperplanes and 
the associated normal cone operators; this corresponds to a mildly overdetermined sys- 
tem of linear equations and to resolvents that are projection mappings (P;)/^/. As the aim 
is to find fixed points of the the averaged resolvent ]a, which in this case is the (equally 
weighted) average of the projections {Pi)iei (see © and (|T5)) ), we measured performance 
at the n iteration of x„ G X by the relative error function in decibel (dB), i.e., by 



For all experiments, the starting point xq is the zero vector. We compared three algorithms 
denoted alg(/^), alg(J o R), and alg(T), which correspond to iterating J o R, and T, 
respectively. The last two new algorithms operate in the product space X; thus, we project 
the nth iterate down to X via {xn)iei = {Xn,i)iel ^ Eiei '^i^n,i to compare to alg(/^). The 
random sets (i.e., the hyperplanes) were generated in 5 instances, and the values of (|68)) 
were averaged for each iteration number. These values are plotted in Figure [U 



(68) 





•14 



•16 







20 



40 60 
Iteration index 



80 



100 



Figure 1: Values of the relative error function for the three algorithms. 
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As seen in Figure [H, the new rigorous algorithm alg(J o R) performs better than 
alg(/^), although the performance gain is slight. (Convergence is guaranteed by 
Fact 13 .^(iv) and Theorem 141] fi) ) Furthermore, the new heuristic algorithm alg(T), 



which currently lacks a convergence analysis (see Remark |4.3|) substantially outperf oms 
alg(JoR). 

Let us now list some open problems. 

Remark 4.6 (open problems) Suppose that m > 3. We do not know the answers to the 
following questions. 

Ql: Concerning (|65)) , is it actually true that 

(69) S = FixT = Fix (Jm o R^ o • • • o Ji o R^) ? 

Q2: Can one give simple sufficient or necessary conditions for the convergence of the 
heuristic algorithm, i.e., the iteration of T, when Fix T 7^ 0? 

Q3: Under the most general assumption ((T3l) , we observed convergence in numerical 
experiments of the new rigorous algorithm even though there is no underlying 
theory — see Proposition B.^liiJ and Theorem 14. 1[ Can one provide simple sufficient 



or necessary conditions for the convergence of the sequence defined by (|59|) ? 

Remark 4.7 Concerning Remark |4.6l we note that the first two questions posed have af- 
firmative answers when m = 2. Indeed, one then computes 

(70) T: X ^ X: (xi,X2) ^ {h-^ a,^2, h-i ^-i j,^X2) 

and hence (xi,X2) ^ FixT if and only if xi = }j^-i^^X2 and X2 = Jj^-ij^^^i, which is the 

same as requiring that (xi, X2) ^ S. When S = FixT 7^ 0, then the iterates of T converge 
weakly to a fixed point by Il30l Theorem 5.3(i)]. 



Appendix A 

Most of this part of the appendix is part of the folklore; however, we include it here for 
completeness and because we have not quite found a reference that makes all points we 
wish to stress. 

We assume that m G {1,2, . . .}, that I = {1,2, . . .,m} and that {Ci)i(^i is a family of 
closed hyperplanes given by 

(71) (Vz G 7) Q- = (x G X I {ui, x) = hi], where fl; G X \ {0} and hi G R, 
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with corresponding projections P/. Set A: X ^ R"^: x i— > (fl/, x) and h = {bi)i(^i G R'". 
Then A* : X: (yi)iei ^ HieiVi'^i- Denote, for every i G I, the z*'^ unit vector in R"^ 

by €{, and the projection R'" R"^ : y ^ {y, e,) onto R by Q,. Note that 

(72) E Q, = Id and (V(z,;) G J x J) Q,Q^- = | = 

10, otherwise. 

We now assume that 

(73) (VzGl) \\ar\\=l, 

which gives rise to the pleasant representation of the projectors as 

(74) (Vz G 7) Pi-.x^x- A*Q,(Ax - h) 
and to (see ©) 

(75) (VxgX) ||Ax-fcf = (fl,,x) = ^||x-P;xf . 

Now let X G X. Using (f74)t and (|72)) , we thus obtain the following characterization of fixed 
points of averaged projections: 



(76a) X G Fix [ J]A, P, 

(76b) <^x = 5^A;P,x 

(76c) ^x = Y^Xi{x- A*Qi{Ax-h)) 
iei 

(76d) ^ X = (j^^^ix) - A* Y^\iQi{Ax - b) 

iei iei 

(76e) ^ A* ( E'^^Q^) (^^ - = 



iei 



iei iei iei iei 

* 







){Ax- 


lel 






m 




)Ax = 



(76h) ^((j^y%Q,)A) ('(Ev^Q,)a)x= ((Ev^Qz)^) (Ev^Q«> 

V / V !6-f / ^ iei ' iei 

(76i) ^ X satisfies the normal equation of the system 



(76j) 



^i6J / 16/ 
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(76k) ^(^(gv^Q.)^) ((Ev^Q.)a' 

(761) = ((EV^tQt)A (LV^tQt)b 

\ iel / iel 

(76m) <=4> X satisfies the normal equation of the system 
(76n) ((Ev^QzVV= (Ev^Q.)^- 

Note that when (A,);^/ = (^)fei, i.e., we have equal weights, then (|76l) and (|72|) yield 

(77a) x.Hx(iEP,)^((EQ,»'((EQ.)^).= ((EQ.)^)*(Sa> 
(77b) ^ A* Ax = A*b 

(77c) X satisfies the normal equation of the system Ax = b 

(77d) <^ X is a least squares solution of the system Ax = b. 

In other words, the fixed points of the equally averaged projections onto hyperplanes are precisely 
the classical least squares solutions encountered in linear algebra, i.e., the solutions to the classi- 
cal normal equation A* Ax = A*b of the system Ax = b. The idea of least squares solutions 
goes back to the famous prediction of the asteroid Ceres due to Carl Friedrich Gauss in 
1801 (see US} Subsection 1.1.1] and also H22l Epilogue in Section 4.6]). 

Example. Consider the following inconsistent linear system of equations 

(78a) x = l 

(78b) x = 2, 

which was also studied by Byrne ||T2l Subsection 8.3.2 on page 100]. Here m = 2 and (|73l) 
holds, and the above discussion yields that Fix (^Pi + and the set of least squares 
solutions coincide, namely with the singleton { | } . Now change the representation to 

(79a) 2x = 2 

(79b) X = 2, 



so that (|73|) is violated. The set of fixed points remains unaltered as the two hyperplanes 
Ci and C2 are unchanged and thus it equals { | } . However, the set of least squares so- 
lutions is now { I } . Similarly and returning to the first representation in (|78l) , the set of 
fixed points will changes if we consider different weights, say Ai = ^ and A2 = §: indeed, 
we then obtain Fix ( + JP2) = { f } while the set of least squares solutions is still { |}. 
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Appendix B 



The proof of the following result is simple and hence omitted. 
Lemma B.l Let {o<-i,j)[i^j)eixi ^^id iri R'"^'", and define 

(80) A : X ^ X : (xf ) e j ^ ( ^ Oii^jX^ and B : X ^ X : (x,) ^ j ^ ^ ■ 
Then 

(81) A o B: X ^ X: {x,)ei ^ ( E'^'J^;) ' 

where G 7 x 7) ^ = Efcel ^!,fc/^fc,j- Furthermore, the following hold: 

(i) If for every i G 7, j a, y = 1 = |6/j, then ^-^j 7; y = 1 as well. 

(ii) If for every ^ I x I, oti^j > and j6; y > 0, then 7, y > as well. 

Proof of Remark |4.3i No R^t is nonexpansive and neither is R^ o • • • o R2 o R^. 

Proof. Take x = {xi)i^i G X, and let i G 7. If i 7^ k, then (Rjtx); = x,; otherwise, i = k 
and (Rjtx)jt is a convex combination of the vectors {xj}j^i-^^]^y In either case, (Rfcx)^ is a 
convex combination of the vectors {Xj}yg7^|jt}- Thus if w G X satisfies \\u\\ =1 and 



(82) (Vz G 7) Xi 



II, if z 7^ k; 
0, if z = k, 



then RjjX = (iz),ej and hence ||Rfcx|P = Hiei \\u\\^ = m > m — 1 = Yljei^{k} Ik' IP = II ''IP- 
Therefore Rjt is not nonexpansive. 

Now assume that k = 1 and that x is defined as in (|82l) . The above reasoning shows that 
Rix = {u)i^i. In view of Lemma B.l, {u)i^j G Fix(R„/ o • • • o R2). Hence (Rm o • • • o R2 o 
Ri)x = {u)iei and thus once again ||(Rm o • • • o R2 o R;[)x|p = m > m — 1 = ||x|p. This 
completes the proof. ■ 

Proof of Remark Let x = (x/);^/ and y = {yi)iei be in X, and take k ^ 1. Using that 
^ir^Ak (flrnily) nonexpansive in (|83c|l , and that || • |p is convex in (|83e[) , we obtain 

(83a) ||(jfcoRfc)x- (jfcoRfc)yf 
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(83b) 

(83c) 

(83d) 
(83e) 
(83f) 



< 



/6Jn{/c} rk 

A; 



A; 



2 

2 



2 



^ E ^h-yif+ E 

j€J\{fc} rk 



Ay + f^jt I 



'i-yj\ 



jel^k} 

i2 



Since (A;).^^ = (^).^^, we further deduce that 

(84a) ^ ^ ^ ^ ,,2 

(84b) 
(84c) 



fcoRfc)x- (jfcoRfc)yll < E 

76JN{fc} 



m — 1 

2 



^7 -1/7 1 
2 



m — 1 



|x-y| 



which implies that Jjt o R^^ is Lipschitz continuous with constant ^Jm/{m — 1). The rest of 
Remark 14.41 now follows from elementary calculus. ■ 
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